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DESCRIPTION 
PARALLEL PULSE SIGNAL PROCESSING APPARATUS^ 
PATTERN RECOGNITION APPARATUS, AND 
IMAGE INPUT APPARATUS 

5 

TECHNICAL FIELD 

The present invention relates to a parallel pulse 
signal processing circuit or neural network having a 
pattern recognition function. 

10 

BACKGROUND ART 

As a scheme for implementing a large-scale neural 
network by small-scale circuits, an arrangement is 
conventionally known which executes time- sharing 

15 multiplexing by using a conventionally disclosed neuron 
model to generate a pulse signal (e.g., Japanese Patent 
Laid-open No. 5-47870 and Japanese Patent No. 2679730) • 
Another arrangement is also disclosed previously which 
can solve wiring problems by forming a neuron element 

20 by using an element to output a pulse train and a 

primary storage means (e.g.^ Japanese Patent Laid-Open 
No. 6-93249) . 

In the field of image recognition, a pattern 
recognition method {method to execute hierarchical 

25 template matching) is proposed which implements a 
recognition processing algorithm specialized to a 
specific recognition target by using a neural network 
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model whose concept is obtained from the information 
processing mechanism of a living body (e.g., Japanese 
Patent Laid-Operi No. 60-712, and Pukushima & Miyake, 
"1982 Neocognitron: A new algorithm for pattern 
5 recognition tolerant of deformation and shifts in 

position. Pattern Recognition, Vol. 15, pp. 455-469). 

As an attempt to more faithfully adopt an 
information processing mechanism based on the neural 
network of a living body, a neural network model 

10 circuit is proposed which transmits and expresses 

information by a pulse train corresponding to an action 
potential (e.g., Murray et al., 1991 Pulse-Stream VLSI 
Neural Networks Mixing Analog and Digital Techniques , 
IEEE Trans, on Neutral Networks, Vol. 2, pp. 193-204, 

15 Japanese Patent Laid-Open Nos. 7-262157, 7-334478, and 
8-153148, and Japanese Patent Nos. 2624143 and 
2879670). 

As a method of recognizing and detecting a 
specific target by a neural network formed from pulse 

20 train generation neurons, there is a scheme using a 
model of a high order (second or higher order) by 
Eckhorn et al. which is premised on linking inputs and 
feeding inputs (e.g., Eckhorn et al. 1990, Feature 
linking via synchronization among distributed 

25 assemblies: Simulation of results from cat cortex. 

Neural Computation, Vol. 2, pp. 293-307), i.e., a pulse 
coupled neural network (to be abbreviated as PCNN 
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hereinafter) (e.g., U.S. Patent No. 5,664,065, and 
Broussard, et al. 1999, Physiologically Motivated Image 
Fusion for Object Detection using a Pulse Coupled 
Neural Network, IEEE Trans, on Neural Networks Vol. 10, 
5 pp. 554-563). 

Of the neural networks according to the prior 
arts, the arrangement which executes time-sharing 
multiplexing has a problem that the processing time 
increases along with an increase in number of synapse 

10 connections. The remaining arrangements can hardly be 
implemented as an electronic circuit because the 
increase in circuit scale and the wiring problems 
become conspicuous as the number of synapse connections 
increases, and the dynamic range of information (e.g., 

15 weight sum value) to be held increases as signals are 
added through synapse connections. 

DISCLOSURE OF INVENTION 
The present invention has been made in 
20 consideration of the above -described problems, and has 
as its object to solve the wiring problems and reduce 
the circuit scale and power consumption. 

In order to achieve an object of the present 
invention, for example, a parallel pulse signal 
25 processing apparatus of the present invention comprises 
the following arrangement. 

That is, a parallel pulse signal processing 
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apparatus including a plurality of pulse output 
arithmetic elements, a plurality of connection elements 
which parallelly connect predetermined elements of the 
arithmetic elements, and a gate circuit which 
5 selectively passes pulse signals from the plurality of 
connection elements, 

characterized in that the arithmetic element 
comprises ^ 

input means for inputting a plurality of time 
10 series pulse signals, 

modulation processing means for executing 
predetermined modulation processing on the basis of the 
plurality of time series pulse signals which are input, 
and 

15 pulse output means for outputting a pulse signal 

on the basis of a result of modulation processing, and 

the gate circuit selectively passes, of the 
signals from the plurality of connection elements, a 
finite number of pulse signals corresponding to 

20 predetermined upper output levels. 

In order to achieve an object of the present 
invention, for example, a parallel pulse signal 
processing apparatus of the present invention comprises 
the following arrangement. 

25 That is, a parallel pulse signal processing 

apparatus which hierarchically executes a plurality of 
arithmetic processing operations, characterized by 
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comprising: 

a plurality of arithmetic elements wliicti receive 
signals from different layer levels and outputs 
predetermined pulse signals by a predetermined local 
5 receptor field structure; and 

a gate circuit element which selectively passes 
the pulse signals from the plurality of arithmetic 
elements belonging to a predetermined receptor field in 
accordance with a signal level of the pulse signal. 

10 In order to achieve an object of the present 

invention, for example, a parallel pulse signal 
processing apparatus of the present invention comprises 
the following arrangement. 

That is, a parallel pulse signal processing 

15 apparatus including input means for inputting data in a 
predetermined dimension, a plurality of data processing 
means, a gate circuit which selectively passes signals 
from the data processing means, and output means for 
outputting a result of pattern detection, 

20 characterized in that the data processing means 

includes a plurality of arithmetic elements parallelly 
connected by predetermined connection means , 

the arithmetic element included in the data 
processing means outputs a pulse- shaped signal train 

25 representing a detection result of a pattern of a 

predetermined category on the basis of an arrival time 
pattern of a plurality of pulses from predetermined 
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aritlunetic elements input in a predetermined time 
window, and 

the output means outputs the detection result of 
the predetermined pattern in the data on the basis of 
5 the outputs from the arithmetic elements. 

In order to achieve an object of the present 
invention, for example, a parallel pulse signal 
processing apparatus of the present invention comprises 
the following arrangement. 

10 That is, a parallel pulse signal processing 

apparatus including input means for inputting data in a 
predetermined dimension, a plurality of data processing 
means for outputting pulse signals, a gate circuit 
which selectively passes signals from the data 

15 processing means, and output means for outputting a 
result of pattern detection, 

characterized in that the data processing means 
includes a plurality of arithmetic elements parallelly 
connected by predetermined connection means, 

20 the gate circuit selectively passes the pulse 

signals on the basis of signal levels of the pulse 
signals from the plurality of data processing means, 

the arithmetic elements receive a time series 
pulse signal, identify time series pulse signal 

25 patterns of a plurality of classes, and output a 

pulse- shaped signal train unique to an arrival time 
pattern of a plurality of predetermined pulse signals 
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input in a predetermined time window, and 

the output means outputs tlie detection result of 

the predetermined pattern in the data on the basis of 

the outputs from the arithmetic elements . 
5 In order to achieve an object of the present 

invention, for example, a parallel pulse signal 

processing apparatus of the present invention comprises 

the following arrangement. 

That is, a parallel pulse signal processing 
10 apparatus which hierarchically executes a plurality of 

arithmetic processing operations, characterized by 

comprising: 

input means for inputting one of an intermediate 
result of different layer levels and data from a 
15 predetermined memory; 

a plurality of data processing means, having a 
feature detection layer which detects a plurality of 
features from the data input by the input means, for 
outputting pulse signals; and 
20 a timing signal generation circuit, 

the data processing means further comprising 

a plurality of arithmetic elements which receive 
detection signals of the features of different types 
from a layer level of a preceding stage and output 
25 predetermined pulse signals, and 

a gate circuit which selectively passes outputs 
from the arithmetic elements involved in the plurality 
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of predetermined features, 

wherein the arithmetic elements output 
pulse- shaped signals at one of a frequency and a timing 
based on a plurality of input signals from the timing 
5 signal generation circuit and an arrival time pattern 
of a plurality of pulses in a predetermined time 
window . 

In order to achieve an object of the present 
invention, for example, a pattern recognition apparatus 
10 of the present invention comprises the following 
arrangement . 

That is, a pattern recognition apparatus 
characterized by comprising the parallel pulse signal 
processing apparatus • 
15 In order to achieve an object of the present 

invention, for example, an image input apparatus of the 
present invention comprises the following arrangement. 

That is, an image input apparatus characterized 
in that pattern recognition is executed by using the 
20 parallel pulse signal processing apparatus, and input 
control of a predeteamained image signal is executed on 
the basis of the pattern recognition result, 

In order to achieve an object of the present 
invention, for example, a control method of a parallel 
25 pulse signal processing apparatus of the present 
invention comprises the following arrangement. 

That is, a control method of a parallel pulse 
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Signal processing apparatus comprising a plurality of 
pulse output arithmetic elements, a plurality of 
connection elements which parallelly connect 
predetermined arithmetic elements, and a gate circuit 
5 which selectively passes pulse signals from the 
plurality of connection elements, 

characterized in that the arithmetic element 
inputs a plurality of time series pulse signals, 

executes predetermined modulation processing on 
10 the basis of the plurality of time series pulse signals 
which are input, and 

outputs a pulse signal on the basis of a result 
of modulation processing, and 

the gate circuit selectively passes, of the 
15 signals from the plurality of connection elements, a 
finite number of pulse signals corresponding to 
predetermined upper output levels. 

In order to achieve an object of the present 
invention, for example, a control method of a parallel 
20 pulse signal processing apparatus of the present 
invention comprises the following arrangement. 

That is, a control method of a parallel pulse 
signal processing apparatus which hierarchically 
executes a plurality of arithmetic processing 
25 operations, characterized by comprising: 

causing a plurality of arithmetic elements to 
receive signals from different layer levels and output 
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precLetermined pulse signals by a predetermined local 
receptor field structure; and 

causing a gate circuit element to selectively 
pass the pulse signals from the plurality of arithmetic 
5 elements belonging to a predetermined receptor field in 
accordance with a signal level of the pulse signal. 

In order to achieve an object of the present 
invention^ for example, a control method of a parallel 
pulse signal processing apparatus of the present 
10 invention comprises the following arrangement. 

That is, a control method of a parallel pulse 
signal processing apparatus comprising input means for 
inputting data in a predetermined dimension, a 
plurality of data processing means, a gate circuit 
15 which selectively passes signals from the data 

processing means, and output means for outputting a 
result of pattern detection, 

characterized by comprising causing each of a 
plurality of arithmetic elements, which are included in 
20 the data processing means and parallelly connected by 
predetermined connection means, to output a 
pulse- shaped signal train representing a detection 
result of a pattern of a predetermined category on the 
basis of an arrival time pattern of a plurality of 
25 pulses from predetermined arithmetic elements input in 
a predetermined time window, and 

causing the output means to output the detection 
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result of the predetermined pattern in the data on the 
basis of the outputs from the arithmetic elements. 

In order to achieve an object of the present 
invention, for example, a control method of a parallel 
5 pulse signal processing apparatus of the present 
invention comprises the following arrangement. 

That is, a control method of a parallel pulse 
signal processing apparatus comprising input means for 
inputting data in a predetermined dimension, a 
10 plurality of data processing means for outputting pulse 
signals, a gate circuit which selectively passes 
signals from the data processing means, and output 
means for outputting a result of pattern detection, 

characterized by comprising causing the gate 
15 circuit to selectively pass the pulse signals on the 
basis of signal levels of the pulse signals from the 
plurality of data processing means, 

causing a plurality of arithmetic elements, which 
are included in the data processing means and 
20 parallelly connected by predetermined connection means, 
to receive a time series pulse signal, identify time 
series pulse signal patterns of a plurality of classes, 
and output a pulse -shaped signal train unique to an 
arrival time pattern of a plurality of predetermined 
25 pulse signals input in a predetermined time window, and 

causing the output means to output the detection 
result of the predetermined pattern in the data on the 
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basis of the outputs from the arithmetic elements. 

In order to achieve an object of the present 
Invention^ for example, a control method of a parallel 
pulse signal processing apparatus of the present 
5 invention comprises the following arrangement. 

That is, a control method of a parallel pulse 
signal processing apparatus which hierarchically 
executes a plurality of arithmetic processing 
operations, the parallel pulse signal processing 

10 apparatus comprising input means for inputting one of 
an intermediate result of different layer levels and 
data from a predetermined memory, a plurality of data 
processing means, having a feature detection layer 
which detects a plurality of features from the data 

15 input by the input means, for outputting pulse signals, 
and a timing signal generation circuit, 

characterized by comprising, under the control of 
the data processing means, causing a plurality of 
arithmetic elements to receive detection signals of the 

20 features of different types from a layer level of a 

preceding stage and output predetermined pulse signals, 
and 

causing a gate circuit element to selectively 
pass outputs from the arithmetic elements involved in 
25 the plurality of predetermined features, and 

causing the arithmetic elements to output 
pulse -shaped signals at one of a frequency and a timing 
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based on a plurality of input signals from the timing 
signal generation circuit and an arrival time pattern 
of a plurality of pulses in a predetermined time 
window - 

5 In order to achieve an object of the present 

invention, for example, a control method of a pattern 
recognition apparatus of the present invention 
comprises the following arrangement. 

That is , a control method of a pattern 
10 recognition apparatus characterized by comprising the 
parallel pulse signal processing apparatus. 

In order to achieve an object of the present 
invention, for example, a control method of an image 
input apparatus of the present invention comprises the 
15 following arrangement. 

That is , a control method of an image input 
apparatus characterized by comprising executing pattern 
recognition by using the parallel pulse signal 
processing apparatus, and executing input control of a 
20 predetermined image signal on the basis of the pattern 
recognition result. 

Other features and advantages of the present 
invention will be apparent from the following 
description taken in conjunction with the accompanying 
25 drawings, in which like reference characters designate 
the same, or similar parts throughout the figures 
thereof . 
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BRIEF DESCRIPTION OF DRAWINGS 
The accompanying drawings, which are incorporated 
In and constitute a part of the specification, 
5 illustrate embodiments of the invention and, together 
with the description, serve to explain the principles 
of the invention. 

Fig. 1 is a block diagram showing the arrangement 
of the main part of a parallel pulse signal processing 
10 apparatus; 

Fig. 2 is a view conceptually showing the overall 
arrangement of a network; 

Fig. 3A is a block diagram showing the 
arrangement of a gate circuit 5; 
15 Pig. 3B is a block diagram showing the 

arrangement of a gate circuit 500; 

Fig. 4 is a circuit diagram showing the 
arrangement of a synapse circuit; 

Fig. 5 is a block diagram showing the arrangement 
20 of the main part of a parallel pulse signal processing 
apparatus according to the third embodiment of the 
present invention; 

Fig. 6 is a block diagram showing the arrangement 
of a gate circuit according to the second embodiment of 
25 the present invention; 

Fig. 7A is a view conceptually showing the 
connection relationship between synapse circuits and 
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neuron circuits; 

Fig. 7B is a block diagram conceptually showing 
the connection relationship between a synapse circuit 
and a neuron circuit; 
5 Fig. 8A is a view schematically showing pulse 

signal propagation from a feature integration layer to 
a feature detection layer (e.g., from a layer (2,0) to 
a layer (1,1) in Fig . 2 ) ; 

Fig. 8B is a timing chart schematically showing a 
10 state in which postsynapse signals are parallelly 
time -window- integrated (a plurality of time window 
integration operations are executed in almost the same 
time zone) in parallelly executing time window 
integration of pulse signals which have passed through 
15 the gate circuit 5; 

Fig. 8C is a view showing inter-neuron connection 
according to the second embodiment; 

Fig. 8D is a timing chart showing the tdLming of 
purlse propagation from feature integration neurons n^, 
20 n^, and n^ (the neurons express features of different 
kinds) on a layer (2,k) to a feature detection neuron 
(n*^) (the neuron executes upper feature detection) on 
a layer (l,k+l) after a timing signal is input to a 
feature integration layer neuron; 
25 Fig. 9 A is a block diagram showing the 

arrangement of a learning circuit; 

Fig. 9B is a block diagram showing the 
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arrangement of a synapse circuit; 

Fig, lOA is a block diagram showing the 
arrangement of a pulse phase modulation circuit; 

Fig. lOB is a timing chart showing the timings of 
5 a square wave PI input to a monostable multivibrator 
506, a square wave P2 output from the monostable 
multivibrator 506, and a square wave P3 output from a 
monostable multivibrator 507; 

Fig. IIA is a timing chart showing a plurality of 
10 pulses corresponding to different feature elements 
which are input to the feature detection neuron in 
detecting high-order graphic information by a pulse 
internal pattern used in the second embodiment; 

Fig. 1 IB is a timing chart showing an example of 
15 a weight function distribution in processing the 

plurality of pulses corresponding to different feature 
elements which are input to the feature detection 
neuron in detecting high- order graphic information by 
the pulse internal pattern used in the second 
2 b embodiment ; 

Fig. lie is a view showing graphic features 
(feature elements) to which the feature detection 
neuron to detect a triangle as a feature reacts, 
including L-shaped patterns (f^^, f^2' • • • ) having various 
25 directions, combination patterns i^^x' ^22' •••^ line 
segments having continuity (connectivity) to the 
L-shaped patterns, and combinations (f^^,...) of parts 
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of two sides included in triangles; 

Fig. 1 ID is a timing chart showing the time 
transition of a weight sum value which is obtained by 
executing weighted summing by pulse input shown in 
5 Pig. IIA and the weight function shown in Fig. IIB; 

Fig. 12 is a block diagram showing an arrangement 
example of a photographing apparatus which includes, as 
a pattern recognition (detection) apparatus, a parallel 
pulse signal processing means according to the first 
10 embodiment of the present invention; and 

Fig. 13 is a graph schematically showing a state 
in which the gate circuit selectively passes, of 
feature integration layer outputs, a maximtmi upper 
signal in each feature class. 

15 

BEST MODE FOR CARRYING OUT THE INVENTION 
Preferred embodiments of the present invention 
will now be described in detail in accordance with the 
accompanying drawings. 
20 [First Embodiment! 

< Outline of Overall Arrangement > 

Fig. 1 shows the arrangement of the main part of 
a parallel pulse signal processing apparatus. The 
parallel pulse signal processing apparatus includes, as 
25 main constituent elements, a data input control circuit 
1, neuron array circuit block 2, synapse array circuit 
block 3, processing result holding memory 4, gate 
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circuit 5, signal generation circuit 6, and overall 
control circuit 7. 

Referring to Fig. 1, the data input control 
circuit 1 is a control circuit to input image data or 
5 the like from a sensor or database and incorporates a 
primary memory. In the neuron array circuit block 2, a 
plurality of neuron circuits belonging to a 
predetermined layer in a hierarchical processing 
structure as shown in Fig. 4 are arrayed. In this 

10 embodiment, one layer (or a neuron involved in 
detecting one feature class in one layer) of a 
multilayered neural network is implemented in an 
arbitrary time zone by using the neuron array circuit 
block 2 and synapse array circuit block 3. A neuron 

15 belonging to another layer (or a neuron involved in 
detecting another feature class) is implemented in 
another time zone. 

In the synapse array circuit block 3, synapse 
connection circuits between neurons are arranged in a 

20 two-dimensional array. A case in which synapse 

connections are present between different layers will 
be described. The synapse array circuit block 3 
implements a synapse connection to at least one layer 
level • 

25 For the synapse array circuit block 3, the 

connection structure to a neuron circuit in the neuron 
array circuit block 2 is controlled by the overall 
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control circuit 7. Ttie overall control circuit 7 
stores receptor field, structure data corresponding to a 
feature class in the internal memory (not shown). The 
receptor field structure data will be described later. 
5 The processing result holding memory 4 temporarily 

holds the output from the neuron array circuit block 2. 

The signal generation circuit 6 supplies the 
output signal from the neuron array to the synapse 
array circuit block 3. The overall control circuit 7 
10 controls the operations of the respective circuit 
blocks to control signal input /output from a lower 
layer to an upper layer in the multilayered neural 
network. That the data which stores the receptor field 
structure data is stored in the internal memozy o£ the 
15 overall control circuit 7 does not provide the major 
function of this embodiment. 

Hence, the means for storing the receptor field 
structure data may be implemented by an arrangement 
different from that shown in Fig. 1, and for example, 
20 an external memory of the parallel pulse signal 
processing apparatus or a memory added to the 
arrangement shown in Fig. 1 as a part of the parallel 
pulse signal processing apparatus. 

Fig. 2 is a view conceptually showing the overall 
25 arrangement of a network implemented by the parallel 

pulse signal processing. When the multilayered neural 
network conceptually shown in Fig. 2 should be mounted 
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as an analog- digital mixed or merged circuit, it is 
often difficult to implement parallel processing by 
parallelly arranging all the constituent elements such 
as neurons and synapses on a two-dimensional plane. In 
5 this embodiment, time- sharing processing is executed by 
using a few neurons or synapse circuit elements. 

The network shown in Fig. 2 mainly processes 
information necessary for recognizing and detecting a 
target (or its geometrical feature) given as an image. 

10 The basic structure of the network is a so-called 

convolutional network structure (reference 1: LeCun, Y* 
and Bengio, Y., 1995, "Convolutional Networks for 
Images Speech and Time Series", in Handbook of Brain 
Theory and Neural Networks (M. Arbib. Ed.), MIT press, 

15 pp. 255-258) having a hierarchical structure. 

This embodiment is different from the 
conventional basic structure in that a synapse 
connection circuit and a gate circuit (to be described 
later) are used for the interlayer connection between a 

20 feature detection layer and a feature integration layer 
(to be described later). The final output (uppermost 
layer output) of this network is a recognition result, 
i.e., the category of the detected target and position 
information on the input data. The outline of each 

25 element shown in Fig. 2 will be described below. 

In an image sensor means, the data input layer 1 
includes a photoelectric conversion element such as a 
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CMOS sensor or CCD element. The feature detection 
layer (1,0) detects low-order features from input image 
data and detects, at each position all over the window 
(or at each of predetermined sampling points all over 
5 the window), local low-order features (the features may 
include color component features in addition to 
geometrical features) of the image pattern equal in 
number to a plurality of feature categories at a 
plurality of scale levels or resolutions at the same 

10 position by multiple resolution processing such as 
Gabor wavelet transform. 

Each neuron element of the feature detection 
layer has a receptor field structure corresponding to 
the type of the feature amount (e.g., the tilt of a 

15 line segment as a geometrical structure when a line 

segment in a predetermined direction is detected as a 
geometrical feature) and generates a pulse train 
corresponding to its degree. 

Feature detection layers (l,k:) form a processing 

20 channel for a plurality of resolutions (or scale 

levels) as a whole (k ^ 0). Assume that Gabor wavelet 
transform is executed by the feature detection layer 
(1,0). In this case, a set of feature detection 
neurons which have, in their receptor field structures, 

25 Gabor filter kernels having the same scale level and 

different directional selectivities foran one processing 
channel in the feature detection layer (1,0). Even in 
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a subsequent feature detection layer (1,1), feature 
detection neurons (the neurons detect higher-order 
features) which receive the outputs from those feature 
detection neurons belong to the same channel as the 
5 processing channel. 

For a detailed method of executing Gabor wavelet 
transform in a neural network, see reference 2: Daugman 
(1988), "IEEE Trans, on Acoustics, Speech, and Signal 
Processing, Vol. 36, pp. 1169-1179". 

10 Even in another subsequent feature detection 

layer (l,k) (k > 1), feature detection neurons which 
receive the outputs from a plurality of feature 
integration neurons which form the same channel in a 
feature integration layer (2,k-l) belong to the same 

15 channel. In each processing channel, processing 

progresses at the same scale level (or resolution) . 
Detection and recognition of low- to high-order 
features are executed by hierarchical parallel 
processing. 

20 Features detected by different processing 

channels at the same layer level (detected features 
have the same complexity level) belong to the same 
category, although they are detected at different 
scales. For example, consider an "eye" pattern in a 

25 face as a middle-order feature. "Eyes" having 
different sizes are detected by the respective 
channels. That is, the plurality of processing 
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Channels having different scale level selectivities 
attempt to detect the "eye" with a given size in the 
image . 

A feature integration layer will be described 

5 next . 

In feature integration layers (2,0), (2, 
each neuron element receives the input from the feature 
detection layer (1,0). The neuron element has the 
above -described predetermined receptor field structure 

10 (the receptor field will indicate the connection range 
to the output element of the immediately preceding 
layer, and the receptor field structure will indicate 
the distribution of the connection weight hereinafter) 
for sub- sampling. More specifically, the feature 

15 integration layer executes arithmetic processing such 

as local averaging by integrating a plurality of neuron 
element outputs in the same receptor field from the 
feature detection layer (1,0) or representative value 
extraction (sub- sampling) by detecting a maximum value 

20 in a local region. 

The receptor fields of image feature portion 
neurons in a feature integration layer have a structure 
which is common to the neurons in that same layer. 
Each of the feature detection layers (1,1), (1,2),..., 

25 (1,N), and feature integration layers (2,1), (2,2),..., 
(2,N) has a predetermined receptor field structure. 
Like the above ^described layers, the former ((1,1),...) 



wo 2005/088535 



- 24 - 



PCT/JP2005/005297 



detects a plurality of different features in the 
feature detection modules . The latter ((2,1),...) 
integrates the detection results for the plurality of 
features from the feature detection layer of the 
5 preceding stage. The feature detection layers are 

connected (wired) to receive the neuron element outputs 
of preceding feature integration layers belonging to 
the same channel. 

The gate circuit 5 will be described next. 

10 The gate circuit 5 selectively passes, of the 

outputs from the plurality of neuron elements in same 
layer, a pulse signal with a high output level. A gate 
circuit which selectively passes the output from a 
feature integration layer is related to a feature 

15 detection layer neuron of the subsequent stage. This 
gate circuit selectively passes a signal of an upper 
output level of the feature integration layer neuron 
outputs for each low-order feature class received by 
the feature detection layer neuron. The gate circuit 

20 function of selectively passing an output from a 

feature detection layer to a feature integration layer 
is implemented by the sub- sampling function executed by 
the feature integration layer neuron of the subsequent 
stage without using the gate circuit 5 shown in Fig. 1. 

25 The gate circuit 5 will be described below 

assuming that it selectively passes an upper-level 
output signal of the output signals from the feature 
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integration layer to the feature detection layer (the 
signal may take the maximxim value of the output 
distribution and have the upper level of the maximum 
value) . 

5 The gate circuit 5 detects the signal levels of 

pulse signal outputs from the feature integration layer 
neurons which belong to a predetermined feature class 
(lower-level) in the receptor field of the feature 
detection layer neuron and passes k (k is a natural 

10 number) upper output level signals (e.g.^ from the 

maximum level to kth level from top) per feature class. 

When the feature detection layer neuron needs to 
detect a plurality of features (their positions on the 
input data are different) belonging to the same feature 

15 class, a given number obtained by multiplying the 

necessary number of local features by k or a number 
close to the given number is used. In addition, the 
gate circuit has a function of selecting an upper 
signal of output signals which take the maximum value 

20 of the feature integration layer neuron outputs of that 
feature class. This function can be implemented by a 
predetermined WTA (Winner Take All) circuit (e.g., 
reference 3: G. Indiveri (2001), "A Neuromorphic VLSI 
Device for Implementing 2-D Selective Attention 

25 Systems", IEEE Trans, on Neural Networks, Vol. 12, 

pp. 1455-1463). In this case, a signal of peak level 
of clusters obtained by feature integration layer 
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neuron outputs belonging to a specific feature class is 
selected. The maximum value or a signal near it may be 
selected. 

In this embodiment, at least one gate circuit 
5 having the above -described function is set between the 
feature integration layer and the feature detection 
layer of the subsequent stage for each feature class 
present in the feature integration layer. 

Fig. 13 schematically shows a state in which for 
10 outputs from jth and (j + Dth feature classes of the 
feature integration layer, gate circuits 5j and 5j+l 
corresponding to these feature classes selectively pass 
top two maximum value signals from the output 
distribution in each class. The selected portions are 
15 indicated by dotted lines in Fig. 13. 

Fig. 3A shows the arrangement of the gate circuit 
5. The gate circuit 5 includes a signal selection unit 
50, clock signal input unit 51, counter 52, gate switch 
53, and gate control circuit 54. The signal selection 
20 unit 50 and clock signal input unit 51 receive signals 
from the layer of the preceding stage. 

Of signals from the feature integration layer 
neurons belonging to the receptor field of the feature 
detection layer neuron corresponding to the gate 
25 circuit 5, only a signal which satisfies the 

above -described predetermined criterion is selectively 
received by the signal selection unit 50. For example. 
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when the outputs from the feature integration layer 
neurons of the preceding stage are temporarily stored 
in the memory, a signal is selectively received by 
appropriately setting the memory address at which each 
5 neuron output belonging to the receptor field is stored 
(e.g., top k output signals are selected). A 
description will be done on the basis of this 
assumption. 

The counter 52 of the gate circuit 5 counts the 

10 number of pulse signals selectively passed. The 

counter 52 is reset by the timing signal from the clock 
signal input unit 51. When the value of the counter 52 
reaches a predetermined upper limit value, the gate 
control circuit 54 opens the gate switch 53, so no 

15 pulse signals are output to the layer of the subsequent 
stage from this time. 

When parallel signal input to a feature detection 
layer should partially be executed in a time -sharing 
manner (for example, signals from a feature integration 

20 layer for receptor fields corresponding to 1/4 of all 
feature detection neurons which detect a predetermined 
feature class are input at once) , the control circuit 
selects a parallelly inputtable number of feature 
detection layer neurons whose receptor fields do not 

25 overlap. In this case, next, the signal selection unit 
50 of the gate circuit 5 sets the memory address to 
store the feature integration layer neuron output 
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corresponding to the receptor field of each feature 
detection layer neuron and reads each output data. In 
the input step after the above -described parallel 
input, signals are sequentially input from receptor 
5 fields different from those input in the preceding 
input step while holding the condition that the 
receptor fields do not overlap each other. 

When the number k of signals to be passed is 1^ 
the signal selection unit 50 of the gate circuit 

10 generally has the function of a so-called winner take 
all (WTA) circuit. When the output from the feature 
integration layer is a phase -modulated pulse signal, 
the signal level is detected on the basis of the delay 
amount from the reference time. The reference time is 

15 given by input of a clock pulse signal from a timing 
element in the overall control circuit 7 . 

The gate circuit 5 is not limited to the 
arrangement shown in Fig. 3A and only needs to have the 
above -described signal selection function. In the 

20 above description, one gate circuit is set for one 

feature class. Upper-level output signals (they may be 
maximum value signals) from a plurality of feature 
classes of the feature integration layer neuron may 
selectively be passed at once. In this case, a signal 

25 which is an upper-level signal in a feature class but 

is not an upper- level signal in all of the plurality of 
classes cannot pass through the gate circuit. 
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A synapse circuit serving as a connection means 
for neuron elements will be described next. 

In this embodiment, the synapse circuit 
implements Interlayer connection. As shown in Fig. 7A, 
5 the synapse circuit includes a signal transmission 

portion (interconnection or delay line) corresponding 
to the axon or dendrite of a neuron and synapse 
connection portions. Fig. 7A shows the arrangement of 
connection portions involved in output (or input when 

10 viewed from the neuron concerned) from neurons (n^) of 
a feature integration (detection) neuron which form a 
receptor field for a certain feature detection 
(Integration) neuron. Referring to Fig. 7A, each 
neuron element n^ outputs a pulse signal train (spike 

15 train) . 

A so-called excitatory connection amplifies a 
pulse signal in the synapse circuit. Conversely, an 
inhibitory connection attenuates a pulse signal. When 
information is transmitted by a pulse signal, 

20 amplification and attenuation can be implemented by 
modulating the amplitude, pulse width, phase, or 
frequency of the pulse signal. 

In this embodiment, when a synapse connection 
portion S is used as a pulse phase modulation element, 

25 amplification of a signal is converted as an actual 
advance of pulse arrival time (or small phase delay 
amount). Attenuation Is converted as an actual delay 
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(or large phase delay amount). That is, qualitatively, 
the excitatory connection gives an advance of phase of 
an arriving pulse with respect to a reference phase, 
and the inhibitory connection gives a delay. As the 
5 synapse connection means, any other modulation method 
such as pulse width modulation or frequency modulation 
can be used (in this embodiment, the description will 
be made by using pulse phase modulation) . The synapse 
connection portions and neuron elements may be put 

10 together to form circuit blocks as shown in Fig. 7B. 

The neuron element included in each layer will be 
described next. Each neuron element serves as a pulse 
signal modulation output element which receives a pulse 
signal and outputs the pulse signal. For example, a 

15 so-called integrate -and- fire neuron which fires and 

outputs a pulse-like signal when a result obtained by 
spatio- temporally linearly adding an input signal (a 
pulse train corresponding to an actual potential) 
exceeds a threshold value, or a nonlinear conversion 

20 circuit which converts the phase or width of a pulse 
signal can be used. 

An image feature portion neuron element of a 
feature detection layer (l,k+l) receives the outputs 
from neuron elements belonging to the plurality of 

25 feature classes of a feature integration layer (2,k). 

On the other hand, a neuron element of a feature 
integration layer (2,k+l) receives the output from a 
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feature detection layer neuron element belonging to the 
same feature class as the neuron element. 

Fig. 4 shows an example of the basic arrangement 
representing the operation principle of the pulse 
5 generation circuit of a neuron element which is formed 
as an integrate~and-f ire circuit. In this example, the 
circuit receives three kinds of inputs, i.e., the 
excitatory and inhibitory inputs from the feature 
integration layer or feature detection layer and a 

10 signal from the timing element in the overall control 
circuit 7. The third input signal is used to supply a 
time window generation timing (timing of the start of 
time integration of the input pulse signal) signal. 

The time constant of a capacitor and resistor 

15 on the excitatory input side is smaller than that of 

capacitor and resistor R^. In the steady state, 
transistors T^, T^, and T^ are OFF. The resistor is 
actually comprised of an active load transistor. 

When the potential of the capacitor increases 

20 and exceeds that of the capacitor by the threshold 
value of the transistor T^, the transistor T^ is 
activated, and then, the transistors T^ and T^ are 
activated. The transistors T^ and T^ form a current 
mirror circuit. The output from the circuit shown in 

25 Fig. 4 is output from an output circuit (not shown) on 
the side of the capacitor C^. VJhen the charge storage 
amount of the capacitor is maximized, the transistor 
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is turned off. As a result, the transistors and 
are also turned off, and the positive feedback becomes 
0. 

In a so-called refractory period, the capacitor 
5 is discharged. Unless the potential of the capacitor 

is higher than that of the capacitor by the 
threshold value of the transistor T^, the neuron does 
not respond. Periodical pulses are output by repeating 
alternate charge /discharge of the capacitors and C^. 

10 The frequency of the pulse is generally determined in 
accordance with the level of the excitatory input. 
However, since the refractory period is present, the 
frequency may be limited to the maximum value, or a 
predetermined frequency may be output. 

15 Instead of using the element having the 

integrate-and-f ire function, a circuit may be used 
which outputs the internal state added with the 
postsynapse pulse signal in the form of a pulse signal 
modulated on the basis of the neuron activation 

20 function (e.g., sigmoid function). 

A time window weight function generation circuit 
serves to control the pulse signal integration 
characteristic in the neuron element . This circuit 
temporally controls the potential of the capacitor (and 

25 charge storage amount). It is weighted stimming in a 

time window (to be described later) for an input pulse 
that reflects the control characteristic . (Figs . IIA to 
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IID) . 

When a synapse weight value corresponds to the 
weight, the time window weight function generation 
circuit generates a weight function corresponding to 
5 the arrival time of the input pulse so that a value 
corresponding to the product between the signal level 
and the synapse weight value is obtained. More 
specifically^ the function waveform is set such that 
the weight value to be multiplied decreases as the 

10 delay of the pulse arrival time increases. To generate 
the function waveform, an analog oscillation circuit is 
used. Alternatively, a digital waveform may be 
generated by a digital circuit using a lookup table 
method or function generation method and converted into 

15 an analog waveform by a D/A converter. 

The reference voltage control circuit generates a 
reference voltage signal (corresponding to the weight 
function in Fig. IIB) on the basis of the input timing 
from the timing element (or cross -coupling input to a 

20 neuron of the subsequent layer) . 

Generally, the relationship between the sum and 
output level (pulse phase, pulse frequency, or pulse 
width) of input signals changes depending on the 
sensitivity characteristic of the neuron. The 

25 sensitivity characteristic can be changed by top-down 
input from the upper layer. For the descriptive 
convenience, assume that the circuit parameters are set 
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such that the frequency of the pulse output 
corresponding to the input signal sum value steeply 
rises (hence, almost binary in the frequency domain), 
and the output level (e.g., the timing for giving phase 
5 modulation) varies by pulse phase modulation. 

Since the reference voltage is controlled by the 
weight function in the time window by the output from 
the image feature portion neuron in the feature 
integration layer, the output characteristic (e.g., the 
10 phase of the pulse output) from the feature detection 
layer neuron changes. VThen phase modulation is 
executed, the phase can be used as the output level of 
the neuron. 

The phase of the neuron output (presynapse) has 
15 such an output characteristic that the initial stage of 
the time window is defined as the reference, and the 
delay (phase) from the reference time is determined by 
the charge storage amount upon receiving a reference 
pulse (output from the timing element or the like) . A 
20 detailed circuit arrangement to give such an output 

characteristic is not the gist of this embodiment and 
will be omitted. When the post synapse pulse phase is 
obtained by adding the presynapse phase to the unique 
phase modulation amount given by the synapse, 
25 multiplication (normal arithmetic processing) of the 

synapse weight and presynapse signal is executed in the 
time window weight function (to be described later) . 
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A known circuit arrangement may be used which 
outputs an oscillation output with a predetermined 
delay when the sum value of inputs obtained by using, 
e.g., a window function exceeds the threshold value. 
5 As described above, the neuron of the feature 

detection layer has a receptor field structure 
corresponding to the feature category. When the weight 
sum value (to be described later) of the input pulse 
signal (current value or potential) from a neuron of 

10 the layer (input layer or feature integration layer) of 
the preceding stage by the time window function is 
equal to or larger than the threshold value, the neuron 
outputs a pulse (the pulse is given by a phase change; 
the frequency, amplitude, or pulse width may be 

15 changed) which takes a value of a nondecreasing 

nonlinear function such as a sigmoid function, i.e., a 
so-called squashing function which asymptotically 
saturates to a predetermined level in accordance with 
the sum value. 

20 <Synapse Circuit Etc.> 

Fig. 4 shows that sub-circuits each of which 
gives a synapse connection strength (the synapse 
connection strength indicates the magnitude of 
modulation related to phase delay) of a neuron ni to 

25 each neuron n"^ as the connection destination in a 

synapse connection portion Si are arranged in a matrix. 
As shown in Fig. 9B, each synapse connection 
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sub -circuit 401 includes a learning circuit 402 and 
phase delay circuit 403. The learning circuit 402 
adjusts the delay amount by changing the characteristic 
of the phase delay circuit 403 and stores the 
5 characteristic value (or control value) in a floating 
gate element or a capacitor connected to a floating 
gate element . 

The phase delay circuit 403 is a pulse phase 
modulation circuit and can include, e.g., monostable 

10 multivibrators 506 and 507, resistors 501 and 504, 

capacitors 503 and 505, and transistor 502, as shown in 
Fig. lOA. Fig. lOB shows the timings of a square wave 
PI input to the monostable multivibrator 506, a square 
wave P2 output from the monostable multivibrator 506, 

15 and a square wave P3 output from the monostable 
multivibrator 507. 

A detailed description of the operation mechanism 
of the phase delay circuit 403 will be omitted. The 
pulse width of the square wave PI is determined by the 

20 time until the voltage of the capacitor 503 by a charge 
current reaches a predetermined threshold value. The 
width of the square wave P2 is determined by the time 
constant by the resistor 504 and capacitor 505. When 
the pulse width of the square wave P2 increases (like 

25 the square wave indicated by a dotted line in 

Fig. lOB), and the trailing edge shifts behind, the 
leading edge of the square wave P3 also shifts by the 
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same amount. However, the pulse width of the square 
wave P3 does not change. Hence, only the phase of the 
input pulse is modulated and output . 

The phase delay amount is given by converting a 
5 phase delay which corresponds to the synapse weight for 
the pulse signal before input to the synapse connection 
and the pulse signal after the synapse connection. In 
this embodiment, the larger the synapse weight value 
becomes , the smaller the phase delay amount becomes . 

10 In place of the phase delay circuit, a time 

window integration circuit (to be described later) 
using a predetermined weight function may be used as 
the synapse connection circuit. In this case, the 
circuit is designed to multiply the pulse signal by a 

15 weight corresponding to the synapse weight. 

When a control voltage is changed by a refresh 
circuit of the reference voltage and the learning 
circuit 402 which controls the charge storage amount in 
a capacitor to give a connection weight, the pulse 

20 phase (delay amount) can be controlled. To hold the 

connection weight for a long time, it may be stored by 
writing it in an analog memory such as a floating gate 
element (not shown) or a digital memory (e.g., SRAM or 
MRAM) after the learning operation. 

25 A circuit having circuit elements shown in 

Fig. 9A is used as an example of the learning circuit 
402 in the synapse, which implements simultaneous 
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arrival of pulses or a predetermined phase modulation 
amount. More specifically, the learning circuit 402 
can include a propagation time measuring circuit 510 
(the propagation time here indicates the time 
5 difference between the pulse output time at the 
presynapse of a neuron of a certain layer and the 
arrival time of the pulse at the output destination 
neuron on the next layer) , a time window generation 
circuit 511, and a pulse shape modulation amount 

10 adjusting circuit 512 which adjusts the pulse shape 
modulation amount in the synapse portion to obtain a 
predetermined propagation time. 

As the propagation time measuring circuit 510, an 
arrangement can be used which receives a clock pulse 

15 from a timing element that forms a single local 

receptor field (to be described later) and obtains the 
propagation time on the basis of the output from the 
counter circuit of the clock pulse in a predetermined 
time width (time window: Fig. 3B) . 

20 <Processing in Feature Detection Layer (1,0) > 

Each neuron of a feature detection layer 10 has a 
receptor field structure corresponding to g^, and g^ 
having the same scale index m has a receptor field with 
the same size. Arithmetically, the corresponding 

25 kernel g^ size also has a size corresponding to the 

scale index. The scales are set in descending order of 
30 X 30, 15 X 15, 7 X 7 on an input image. Each neuron 
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outputs a pulse at an output level (the phase Is used 
as the reference: the frequency, amplitude, or pulse 
width may be used as the reference) as a nonlinear 
squashing function of a wavelet transform coefficient 
5 value obtained by executing product -sum input of the 
distribution weight function and image data. As a 
result, Gabor wavelet transform is executed as the 
output of the entire layer (1,0). 

<Processing in Feature Detection Layer ( Middle -Order , 
10 High-Order Feature Extraction) > 

Subsequent feature detection layers 
((1,1), (1,2),...) detect middle- to high- order image 
features at predetermined sampling positions on the 
input image . Unlike the feature detection layer (1,0), 
15 each neuron forms the receptor field structure unique 
to the feature pattern to be detected by a so-called 
Hebb learning rule. The size of the local region where 
feature detection is executed stepwise approaches to 
the size of the entire recognition target toward the 
final layer so that the middle- or high-order features 
are detected geometrically. For example, for 
detection/recognition of a face, middle-order (or 
high-order) features are features at the graphic 
element level such as eyes, nose, and mouth in the 
face. The feature detection layer neuron can generally 
(independently of low- and high-order feature 
extraction) have a mechanism to receive inhibitory 
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(shunting inhibition) connection on the basis of the 
output from the layer of the preceding layer to 
stabilize the output. 

<Processing in Feature Integration Layer (Image Feature 
5 Portion) > 

The neuron element of a feature Integration layer 
((2,0), (2,1),....) As shown in Fig. 2, connection 
from a feature detection layer (e.g., (1,0)) to a 
feature integration layer (e.g., (2,0)) is designed 

10 such that both the excitatory connection input from the 
neuron element of the same feature element (type) of 
the feature detection layer of the preceding stage in 
the receptor field of the feature integration neuron 
and the timing element output (PNoutl) (to be described 

15 later) are received on the excitatory input side shown 
in Fig . 4 . The neuron element of the integration layer 
is local averaging for each feature category or 
sub-sampling such as maximum value detection, as 
described above. 

20 The feature integration layer neuron receives 

pulses associated with a plurality of features of the 
same type and integrates and averages them in a local 
region (receptor field) (or calculates a representative 
value such as a maximum value in the receptor field) , 

25 thereby properly detecting even fluctuation or 

deformation of the position of the feature. For this 
reason, the receptor field structure of the feature 
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integration layer neuron element can be designed to be 
uniform (for example, the receptor field is a 
rectangular region having a predetermined size in which 
sensitivities or weight coefficients are uniformly 
5 distributed) regardless of the feature category. 
<Operation Principle of Pattern Detection> 
Pulse coding and detection method of 
two-dimensional graphic pattern when the network 
arrangement shown in Fig. 2 is applied to a pattern 

10 recognition apparatus will be described below. Fig. 8A 
schematically shows pulse signal propagation from a 
feature integration layer to a feature detection layer 
(e.g., from the layer (2,0) to the layer (1,1) in 
Fig. 2). The gate circuit 5 is arranged before synapse 

15 connection elements (Sj) to pass only pulse signals (PI 
and P3) significant for feature detection. Fig. 8A 
shows that only pulses modulated by synapse connection 
elements SI and S3 propagate. The respective neurons n^ 
on the feature integration layer side correspond to 

20 different feature amounts (or feature elements). The 
neuron n"^ on the feature detection layer side is 
involved in detection of a higher-order feature 
(graphic element) obtained by combining the features in 
the same receptor field. 

25 On the input side of the synapse connection 

circuits, the gate circuit 5 is set which opens for 
only a pulse signal having the maximum value of the 
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upper level to transmit only the upper level of the 
feature integration layer outputs by connection from 
the feature integration layer neuron to the feature 
detection layer neuron. When a timing pulse signal 
5 (PN) is input, the output pulse signal from the feature 
integration layer neuron undergoes modulation 
processing such as phase modulation by the synapse 
connection circuit and propagates to the next feature 
detection layer. In the feature detection layer 

10 neuron, time integration of the pulse signal which has 
passed through the synapse connection for the feature 
integration layer neuron is executed in the range of 
the time integration window. 

The time window is defined for each feature 

15 detection neuron (n'i) commonly to the neurons in the 
feature integration layer which form the same receptor 
field for the neuron and gives the time range of the 
time window integration. 

Referring to Fig. 8A, the gate circuit 5 is 

20 located before the synapse connections of the feature 
detection layer neurons. The neurons which input 
signals to the gate circuit 5 form the receptor field 
of the feature detection layer neurons. They are all 
feature integration layer neurons belonging to the same 

25 feature class. In this embodiment, the feature 

detection layer neurons parallelly execute time window 
integration of the pulse signals which have passed 



wo 2005/088535 



- 43 - 



PCT/JP2005/005297 



through the gate circuit 5. Fig. 8B schematically 
shows a state in which the postsynapse signals are 
parallelly time-window-integrated (a plurality of time 
window integration operations are executed in almost 
the same time zone) . 

The timing element is eirranged in the overall 
control circuit 7 and outputs a timing pulse signal to 
the feature integration neurons (layer number (2,k-l)) 
and feature detection neurons (layer number (l^k)), 
thereby giving a time window generation timing signal 
when the feature detection neuron temporally adds 
inputs. The start time of the time window is set as 
the reference time to measure the arrival time of the 
pulse output from each feature integration neuron. 
That is, the timing element gives a reference pulse for 
the pulse output time from the feature integration 
neuron and time window integration in the feature 
detection neuron. 

In this embodiment, neurons which correspond to 
low-order feature elements in the same receptor field 
on the feature integration layer as the layer of the 
preceding stage of the feature detection layer and are 
involved in detecting a predetermined local feature in 
the input image synchronously fire (output pulses) in 
predetermined phases. Generally, there is a connection 
from a neuron of the feature integration layer to a 
feature detection neuron which detects identical 
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high- order features at different positions (in this 
case^ although the receptor fields are different^ 
connections which form identical high-order features 
are present). At this time, synchronous firing occurs 
5 even between these feature detection neurons. 

Each pulse passes through the synapse circuit, is 
given with a phase delay in a predetermined amount, and 
arrives at the feature detection neuron through the 
signal transmission line. The sequence of pulses on 

10 the time axis is represented by pulses (PI, P2, and P3) 
indicated by dotted lines on the time axis of the 
feature detection neuron. 

In the feature detection neuron, when a value 
larger than the threshold value is obtained by time 

15 window integration (normally, time window integration 
is executed once; charge storage or averaging by a 
number of times of time window integration may be done) 
of each of pulses (PI, P2, and P3), pulse output (Pd) 
is done on the basis of the end time of the time 

20 window. The time window in learning shown in Fig. 8A 
is referred to in executing the learning rule (to be 
described) . 

<Feature Detection Layer Processing> 

Processing (in learning and recognition) mainly 
25 executed in the feature detection layer will be 

described below. In each feature detection layer, as 
described above, in a processing channel set for each 
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scale level, pulse signals associated with, a plurality 
of different features from the same receptor field are 
received, and spatio-temporal weigl:ited summing (load 
summing) and threshold processing are executed. 
5 In each neuron on the feature detectibn layer, 

the spatio-temporal weighted summing (load summing) of 
the input pulse is executed in only a time window 
having a predetermined width for a pulse train which 
has arrived at the neuron. The means for implementing 
10 weighted summing in the time window is not limited to 
the neuron element circuit shown in Fig. 4. The 
weighted summing may be implemented by any other 
method . 

The time window corresponds to a time zone except 
15 the refractory period of the actual neuron to some 

extent. More specifically, in the refractory period 
(time range except the time window), no output from the 
neuron is executed regardless of the input. However, 
in the time window except that time range, firing 
20 corresponding to the input level is done, like the 
actual neuron. 

In this embodiment, the timing pulse signal from 
the timing element (pulse output at a fixed frequency) 
in the control circuit is input for all processes in 
25 each layer, thereby implementing commonality between 
the neurons in the above -described time window start 
period. In the arrangement shown in Fig. 1, when the 
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synapse array circuit block 3 and neuron array circuit 
block 2 form one layer of the entire hierarchical 
structure shown in Fig. 2, the timing signal Is 
commonly used In the layer. Even when the synapse 
5 array circuit block 3 and neuron array circuit block 2 
form a plurality of layers corresponding to part of the 
entire hierarchical structure shown In Fig. 2, time 
window synchronization control need not be executed 
throughout the network (even If It Is necessary) . The 
10 problem of skew of the timing signal Is relaxed as a 
whole . 

<Appllcatlon Example In Which Apparatus Is Mounted In 
Image Sensing Apparatus or Robot Visual Head> 

When the parallel pulse signal processing 

15 apparatus according to this embodiment is mounted, as a 
pattern recognition (detection) apparatus, in a robot 
visual head or an image sensing apparatus having an 
optical-axis direction control mechanism, a specific 
object in the visual field can be detected at a high 

20 speed, and optimum photography (auto-focusing or color 
correction or exposure of a specific object) control 
can be executed. The arrangement in this case will be 
described with reference to Fig. 12. 

An image sensing apparatus 9 shown in Fig. 12 

25 comprises an imaging optical system 92 which includes a 
photographing lens and a driving control mechanism for 
zoom photography and optical-axis direction control, a 
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CCD or CMOS image sensor 93, an linage sensing parameter 
measuring unit 94, a video signal processing circuit 
95, a storage unit 96, a control signal generation unit 
97 for generating a control signal to control the image 
5 sensing operation or image sensing conditions, a 

display 98 also serving as a viewflnder such, as EVP, an 
electronic flash unit 99, a recording medium 910, and a 
gaze recognition processing unit 911 (pattern 
recognition apparatus including the above -described 

10 parallel pulse signal processing circuit having the 
hierarchical structure of this embodiment). 

In this image sensing apparatus 9, the direction 
of presence of a person registered in advance is 
estimated from an image signal input from the image 

15 sensor 93. The optical axis of the imaging optical 
system 92 is controlled to that direction. The face 
image (the position of presence and size) is detected 
by the gaze recognition processing unit 911. The 
pieces of information of the position (presence 

20 direction) and size of the person are input to the 

control signal generation unit 97. The control signal 
generation unit 97 generates control signals to 
optimally control focusing, exposure condition, and 
white balance for the person on the basis of the output 

25 from the image sensing parameter measuring unit 94. 

When the parallel pulse signal processing 
apparatus according to the above-described embodiment 
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is used in the image sensing apparatus which detects 
(recognizes) a pattern from an image, the object 
detection (recognition) function can properly be 
implemented at low power consumption and high speed 
5 (real time) . Hence, detection of a person or the like 
and optimxam control (AF or AE) of photography based on 
it can be done. 
[Second Embodiment] 

In this embodiment, a pulse signal to a neuron 
10 element is input through a common bus line to propagate 
a bundle of signals after synapse connection. For a 
feature detection layer neuron, of the outputs from the 
feature integration layer of the preceding layer, 
pulses which have undergone phase modulation unique to 
15 the feature class are time-serially input. A gate 

circuit 5 is set after synapse connection circuits on 
the common bus line where the synapse connections 
converge . 

Fig. 6 shows the arrangement of the gate circuit 
20 used in this embodiment. The gate circuit 5 includes a 
signal selection unit 50, clock signal input unit 51, 
gate switch 53, gate control circuit 54, and delay time 
based comparator 55. The signal selection unit 50 
accesses a processing result holding memory 4 to 
25 extract a signal from the receptor field. The signal 
selection unit 50 has no so-called WTA function. 

On the other hand, the comparator 55 is a circuit 
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to select only upper-level signals (or upper-level 
maximum value signals) on the basis of the delay amount 
of the pulse arrival time with respect to the reference 
time given from a timing element. The comparator 55 
can be regarded as a WTA circuit in the time -axis 
domain. The functions of the remaining constituent 
elements are the same as those in the arrangement shown 
in Fig. 3A described in the first embodiment. 

Referring to Fig. 8C, in each inter-neuron 
connection, a unique delay (unique to the feature) is 
generated due to the pulse propagation time and the 
time delay in the synapse connection (S^^,) from a 
neuron n^ to a neuron n'^. As a result, a pulse train 
arriving at the neuron n'^ has a predetermined order 
(and interval) on the basis of the delay amount in the 
synapse connection determined by learning as long as a 
pulse is output from each neuron of the feature 
integration layer (Fig. 8C shows that pulses arrive in* 
the order of P4, P3 , P2, and PI). As described above, 
the gate circuit 5 selectively passes the pulse signals 
of the pulse train which are modulated by the synapse 
connections and propagate on the common bus. 

Fig. 8D shows the timing of pulse propagation 
from feature integration neurons n^, n^, and (the 
neurons express features of different kinds) on a layer 
(2,k) to a feature detection neuron (n'p (the neuron 
executes upper feature detection) on a layer (l,k+l) 
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after a timing signal is input to a feature integration 
layer neuron. 

Calculation processing of the internal state of 
the feature detection layer neuron is executed by 
5 predetermined weighed time window integration. 

Referring to Fig. 8C, in the feature detection layer 
neuron, time window integration to be described later 
is not executed before input from the timing element. 
The integration is triggered by pulse input from the 

10 timing element. 

The refractory period shown in Fig. 8D is the 
time zone immediately after firing of the feature 
detection neuron until the next time window start time. 
The length of the refractory period and the width of 

15 the time window can arbitrarily be set. The refractory 
period need not always be shorter than the time window 
as shown in Fig. 8D. 

Pulses corresponding to feature amounts arrive at 
a predetermined time interval due to a delay amount 

20 (phase) defined in advance by learning. Learning 

control of the pulse arrival time jpattern is not the 
gist of this embodiment, and a detailed description 
thereof will be omitted. For example, competitive 
learning is introduced so that a pulse for a feature 

25 element of a graphic pattern, which most contributes to 
detection of the graphic, arrives early. For feature 
elements whose pulse arrival times can almost the same. 
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tlie pulses arrive at a predetermined interval. 
Alternatively, the pulses may arrive at different time 
intervals for predetermined feature elements (feature 
elements which form the recognition target and can be 
5 regarded as especially important : e.g., features having 
a large mean curvature or high linearity) . 
< Spat io -Temporal Integration of Pulse Outputs and 
Network Characteristic> 

The arithmetic processing of spatio-temporal 

10 weighted summing (load summing) of input pulses will be 
described next. As shown in Fig. IIB, in each neuron, 
the weight sum of input pulses is calculated for each 
sub- time window (for each time slot) by using a 
predetermined weight function. The sum of the weight 

15 sums is compared with the threshold value. In this 

case, represents the central position of the weight 
function in a sub-time window j. It is represented on 
the basis of the time window start time (elapse time 
from the start time) . The weight function is generally 

20 a function of the distance (shift on the time axis) 

from a predetermined central position (representing the 
pulse arrival time when a feature to be detected is 
detected) . 

In the synapse, phase modulation corresponding to 
25 the type of feature is executed. If multiplication of 
the synapse weight value and the signal is not 
executed, in time window integration, multiplication 
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corresponding to ttie synapse weight is executed for the 
neuron output level in the layer of the preceding stage 
by referring to the' weight function. 

Assume that the weight function in each sub -time 
5 window has a symmetrical shape, or the weight function 
is a function of the distance on the time axis . When 
the central position x of the weight function in each 
sub- time window (time slot) of the neuron indicates the 
time delay after learning between neurons, the neural 
10 network which executes the spatio-temporal weighted 

summing (load summing) of input pulses can be regarded 
as a kind of radial basis function network (to be 
abbreviated as RBF hereinafter) of the time axis 
domain . 

15 The weight function may have an asymmetrical 

shape (as shown in Fig. IIB) or take a negative value. 
For example, assume that the neuron of a feature 
detection layer is expected to finally detect a 
triangle, and a feature i^^^^^J which is apparently not 

20 the constituent element of the graphic pattern is 

detected. In this case, to prevent triangle detection 
output from finally being done even when the 
contribution from the remaining feature elements is 
large, connection from a weight function and feature 

25 detection (integration) neuron, which gives negative 
contribution, can be given in advance from a pulse 
corresponding to the feature (F . ) in the input sum 
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calculation processing. 

A spatio-temporal sum X^(t) to tlie neuron n^ of 
the feature detection layer is given by 

Xi(t) = 2 SilFTi(t)yj(t - T^j - Ej) 

5 Where 8^ is the initial phase of the output pulse from 
the neuron n^, F^^ is the time window weight function. 
When convergence to 0 occurs by synchronous firing with 
the neuron n^ or the phase of the time window is 
forcibly synchronized with 0 by timing pulse input, 8^ 

10 can always be 0. 

When weighted summing by the pulse input shown in 
Fig. IIA and the weight function shown in Fig. IIB is 
executed, the time transition of the weight sum value 
as shown in Fig* IID is obtained. When the weight sum 

15 value reaches a threshold value (V^) , the feature 

detection neuron outputs a pulse. The output pulse 
signal from the neuron n^^ is output to the neuron of 
the upper layer with an output level as the squashing 
nonlinear function of the spatio-temporal sum 

20 (so-called total input sum) of the input signals and a 
time delay (phase) given by the learning, as described 
above (the pulse output has a fixed frequency (binary) 
and is output by adding a phase modulation amount as 
the squashing nonlinear function for the 

25 spatio-temporal sum of the input signals to the phase 
corresponding to the fixed delay amount determined by 



wo 2005/088535 



- 54 - 



PCT/JP2005/005297 



leaxning) . 

A feature detection neuron which detects a 
triangle as a feature will be described below for the 
sake of simplicity. The feature integration layer of 
5 the preceding layer reacts to graphical features 
(feature elements) as shown in Fig. IIC such as 
L- shaped patterns (f^^, f^^' • • • ) having various 
directions, combination patterns {^21' ^22' • * • ^ line 
segments having continuity (connectivity) to the 

10 L-shaped patterns, and combinations (fg^,..*) of parts 
of two sides which form a triangle. 

Referring to Fig. IIC, f^^, f^2' ^43 indicate 

features which form triangles with different 
directions, i.e., features corresponding to f^^, and 

15 fj3. When a unique delay amount is set by learning 

between neurons which form interlayer connection, in 
the feature detection neuron for the triangle, in each 
sub- time window (time slot) (wl, w2,...) obtained by 
dividing the time window, setting is done in advance 

20 such that pulses corresponding major and different 
features of the triangle should arrive. 

For example, pulses corresponding to the 
combination of feature sets which form a triangle as a 
whole arrive first at wl, w2,..., wn after the time 

25 window is divided by n, as shown in Fig. IIA. The 

delay amount is set by learning such that the L-shaped 
patterns (f^^^, f^^' ^13^ arrive in wl, w2, and w3, and 
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the pulses corresponding to tiie feature elements (£21' 
f , and f„) arrive in wl^ w2, and w3. 

Pulses corresponding to the feature elements i^^^, 
f , and f„) also arrive in the same order as described 

32 33 

5 above. In the example shown in Fig. IIA^ a pulse 
corresponding to one feature element arrives in one 
sub- time window (time slot). 

Assume that the setting is done in advance such 
that a pulse corresponding to an important feature 
-10 arrives in an early sub -time window. When the weight 
function value in the sub- time window is larger than 
the value in another sub -time window, the significant 
feature can more easily be detected. The significance 
(detection priority of features) can be acquired by 
15 learning or defined in advance. 

The number of divided sub -time windows (time 
slots), the width of each sub- time window (time slot), 
the feature class, and the time interval allocation of 
a pulse corresponding to a feature are not limited to 
20 the above description and can be changed. 
[Third Embodiment] 

In this embodiment, on the basis of the 
distribution of synapse connections which form the 
receptor field of a feature detection layer neuron, a 
25 gate circuit selects a synapse connection whose 

intensity is maximum and whose absolute value falls 
within the range of top k (k is a natural number) , and 
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passes only a pulse signal from the synapse connection. 

Fig. 5 shows the main part of a parallel pulse 
signal processing apparatus according to this 
embodiment. The parallel pulse signal processing 
5 apparatus includes, as main constituent elements, a 
data input control circuit 100, neuron array circuit 
block 200, synapse array circuit block 300, processing 
result holding memory 400, gate circuit 500, signal 
generation circuit 600, and overall control circuit 
10 700. The function of each constituent element is the 

same as in the first embodiment except the gate circuit 
500. 

Unlike the arrangement of the first embodiment, 
the gate circuit 500 accesses the synapse array circuit 

15 block 300 and selects a significant synapse connection. 
A significant synapse connection indicates a synapse 
connection whose intensity is maximtmi and whose 
absolute value falls within the range of top k (k is a 
natural number), as described above. 

20 Fig. 3B shows the arrangement of the gate circuit 

500 which includes, as main constituent elements, a 
signal selection unit 60, clock signal input unit 61, 
and gate control circuit 64. The signal selection unit 
60 has a function of a WTA circuit (see reference 3 in 

25 the first embodiment) to temporarily read out all 

synapse connection data from the internal memory of the 
synapse connection circuit array 300 shown in Fig. 5 
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and extract the weightt values of synapse connections 
corresponding to the maximum values of top k from the 
memory addresses, and selects the address of a 
significant synapse connection. The gate control 
5 circuit 64 selectively passes a pulse signal modulated 
by the synapse connection extracted by the signal 
selection unit in synchronism with a timing signal from 
the clock signal input unit 61. 

As many apparently widely different embodiments 
10 of the present invention can be made without departing 
from the spirit and scope thereof, it is to be 
understood that the invention is not limited to the 
specific embodiments thereof except as defined in the 
cladLms . 

15 
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